<!DOCTYPE html>
<head>
    <title>METviewer Documentation</title>
    <link rel="stylesheet" type="text/css" href="mv_doc.css"/>
    <link rel="shortcut icon" href="include/ral_icon.ico" type="image/x-icon"/>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <script type="text/x-mathjax-config">
    MathJax.Hub.Config({
      tex2jax: {inlineMath: [["$","$"],["\\(","\\)"]]}
    });



    </script>
    <script type="text/javascript" src="MathJax.js?config=TeX-AMS_HTML-full"></script>

</head>
<body>

<p class="loc" style="padding-top:10px">
    <b>Location:</b> <a class="loc" href="index.html">Home</a> &#187; Batch Plotting
</p>
<hr/>
<h2>EMC vs. NCAR method of determining p-values for Aggregation statistics </h2>
<p>In statistics, the p-value represents the probability of a statistic having a value at least as
    extreme as observed, in this case a difference between two
    models, given the null hypothesis that the two models come from the same distribution. When the
    p-values are large, there is evidence to reject this null
    hypothesis.
</p>
<p>
    The EMC and NCAR methods of determining p-values are largely the same, though the output is quite
    different. Both are based on estimates of the mean and
    standard deviation of the statistic. Along with the sample size, the standard Student's t value is
    calculated.\[\overline{x} \pm t_{\alpha/2} \bigl(s /
    \sqrt{n} \bigr)\] When this value is
    compared with the known Student's t distribution, the significance level (i.e. p-value) is
    determined. The first difference between the two methods is the
    identification of the appropriate \[t_{\alpha/2} \] value. In the NCAR version, this value is
    determined exactly using the statistic and the appropriate
    sample size.
    In the EMC version, an approximate lookup table for the Student's t distribution is hard coded.
    Sample sizes are in four groups: &#60; 20 cases, 20-40 cases,
    40-80 cases, and &#62; 80 cases. NCAR output contains the actual probability, while EMC output is
    grouped. Any non-significant differences are denoted by NA,
    significant differences fall into categories of 0.95, 0.99, and 1. Both systems use negative signs
    to denote a better statistic from the second model.
</p>
<p>
    P-values can also be determined by bootstrapping using the percentile method. Cases (e.g. days)
    are randomly selected with replacement, then the difference
    between the statistics for two models is determined based on that sample. This process is repeated
    a large number of times, resulting in an empirical
    distribution of the differences. To find the p-value, the total percent of the bootstrap
    differences that are more extreme than the observed difference are
    calculated. This is the estimated p-value. The only difference here is that the difference is
    compared to a resampled distribution rather than a known,
    theoretical distribution.
</p>

<p>Some of statistics do not have p-values:<br/>
    BASER,FMEAN,FBAR,FSTDEV, OBAR, OSTDEV, FRANK_TIES, ORANK_TIES,
    FBAR, FSTDEV, OBAR, OSTDEV, RANKS, FRANK_TIES, ORANK_TIES,VL1L2_FBAR, VL1L2_OBAR,
    VL1L2_FSTDEV,VL1L2_OSTDEV,VL1L2_FOSTDEV, PSTD_BASER,PSTD_RESOLUTION, PSTD_UNCERTAINTY,PSTD_ROC_AUC
</p>

<h2>METviewer Documentation - Scorecard Module</h2>

<p>The scorecard Module is used to generate a scorecard image and HTML table using data in the
    METviewer database. The usage statement:</p>

<p class="term">
    USAGE: mv_scorecard.sh scorecard_spec_file<br/>
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;where&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"scorecard_spec_file"
    specifies the XML plot specification
    document<br/>
    ---- Scorecard Done ----<br/>
</p>
<p>
    The column fields values can be combine. For example, it possible to combine several vx masks together into one.
    If someone splits vx_mask into the 14 subregions over the CONUS (APL, NEC, SEC, GMC, LMV, MDW, NPL, SPL, NMT,
    SMT, GRB, NWC, SWC, SWD) than he can select all of the regions and the generated plot will
    contain statistics over the entire CONUS. In this case hte name of the field would look like a string with field
    values
    separated by ';'
    <br/>
    APL;NEC;SEC;GMC;LMV;MDW;NPL;SPL;NMT;SMT;GRB;NWC;SWC;SWD
</p>

<p>
    The <span class="term">scorecard_spec_file</span> passed to the scorecard module contains
    information about the scorecard that will be generated. It is an XML
    file whose
    top-level tag is <span class="code">&lt;plot_spec&gt;</span> which contains the following
    elements, divided into functional sections:

</p>
<ul>
    <li><span class="code">&lt;connection&gt;</span> see <a href="common_xml.html">common xml</a></li>
    <br/>

    <li><span class="code">&lt;rscript&gt;</span> specifies the Rscript instance to use, for example
        <span class="term">/usr/local/bin/Rscript</span>,
    </li>
    <br/>

    <li>
        <span class="code">&lt;folders&gt;</span>: a structure that specifies the input and output file
        system folders
        <ul>
            <li><span class="code">&lt;r_tmpl&gt;</span>: the folder that contains the R templates used by
                METviewer, typically <span class="term">R_tmpl/</span>
                from the metviewer
                CVS source tree
            <li><span class="code">&lt;r_work&gt;</span>: the folder that will contain the data and
                generated R scripts, must contain the sub-folder
                <span class="term">include/</span>
            <li><span class="code">&lt;plots&gt;</span>: the output folder that will contain generated
                image and HTML files
            </li>
            <li><span class="code">&lt;data&gt;</span>: the output folder that will contain generated data
                files
            </li>
            <li><span class="code">&lt;scripts&gt;</span>: the output folder that will contain scripts
                files
            </li>
        </ul>
    </li>
    <br/>

    <li>
        <span class="code">&lt;plot&gt;</span>: a structure that specifies scorecard characteristics
        <ul>
            <li><span class="code">&lt;execution_type&gt;</span>: <span class="code">Rscript</span> or
                <span class="code">Python</span>, indicating whether to use Rscript ot Python for the statistics
                calculation and plotting (if available). This is optional element. The default value is Rscript
            </li>
            <li><span class="code">&lt;view_value&gt;</span>: <span class="code">TRUE</span> or
                <span class="code">FALSE</span>, indicating whether or not to
                display statistical values in the scorecard cells. The default value is FALSE (do not
                display)
            </li>
            <li><span class="code">&lt;view_symbol&gt;</span>: <span class="code">TRUE</span> or
                <span class="code">FALSE</span>, indicating whether or not to
                display a symbol in the scorecard cells. The default value is TRUE ( display)
            </li>
            <li><span class="code">&lt;view_legend&gt;</span>: <span class="code">TRUE</span> or
                <span class="code">FALSE</span>, indicating whether or not to
                display a legend for the scorecard. The default value is TRUE ( display)
            </li>
            <li><span class="code">&lt;stat_flag&gt;</span>: <span class="code">NCAR</span> or
                <span class="code">EMC</span>, which algorithm to use to calculate
                statistics.The default value is NCAR
            </li>
            <li><span class="code">&lt;printSQL&gt;</span>: <span class="code">FALSE</span> or
                <span class="code">TRUE</span>, indicating whether or not to
                print SQL queries. The default value is FALSE ( do not print)
            </li>
            <li><span class="code">&lt;stat&gt;</span>: <span class="code">DIFF</span> (display a difference between
                models)
                or <span
                        class="code">DIFF_SIG</span> ( display p_value ) or <span class="code">SINGLE </span>(display
                a value of statistic for the 1st model. In this case the value of the 2nd model is optional. This mode works only for agg_stat=true ),
                indicating which statistic use to display for both - values and symbols - in the cell.
                The default value is DIFF_SIG ( display p_value )
            </li>
            or

            <li><span class="code">&lt;stat_value&gt;</span>: <span class="code">DIFF</span> (display a difference
                between models)
                or <span
                        class="code">DIFF_SIG</span> ( display p_value ) or <span class="code">SINGLE </span>(display
                a value of statistic for the 1st model. In this case the value of the 2nd model is optional. This mode works only for agg_stat=true ),
                indicating which statistic use to display for values in the cell.
            </li>
            <li><span class="code">&lt;stat_symbol&gt;</span>: <span class="code">DIFF</span> (display a difference
                between models)
                or <span
                        class="code">DIFF_SIG</span> ( display p_value ) or <span class="code">SINGLE </span>(display
                a value of statistic for the 1st model. In this case the value of the 2nd model is optional. This mode works only for agg_stat=true ),
                indicating which statistic use to display for symbols in the cell.
            </li>

            <li><span class="code">&lt;threshold_file&gt;</span>: full path to the XML file describing
                configurations for thresholds, colors and symbols (optional)
            </li>
            <li><span class="code">&lt;weight_file&gt;</span>: full path to the XML file describing
                configurations for weights definitions (optional)
            </li>
            <li><span class="code">&lt;left_column_names&gt;</span>: a list of names for the most left columns as
                in &lt;val&gt;NAME&lt;/val&gt; format
            </li>

            <li><span class="code">&lt;symbol_size&gt;</span>: a size for the symbols. It can be in '%' or 'px'.
                For example, '120%' will increase the size of symbols in 1.2 times. '20px' can also be used
            </li>
            <br/>

            <a name="template"></a>
            <li>
                <span class="code">&lt;template&gt;</span>: the name of the R plot script template to use.
                The following templates are currently supported:
                <ul>
                    <li><span class="term">scorecard.R_tmpl</span></li>
                </ul>

            </li>
            <br/>

            <a name="plot_fix"></a>
            <li><span class="code">&lt;plot_fix&gt;</span>: describes database fields - models, date range
                and init hour - that are constant.<br/>
                <br/><b>Attributes:</b>
                <ul style="list-style-image:url('att_bullet.gif')">
                    <li><span class="code">name</span>: the database field name whose values are contained
                        in the child tags
                    </li>
                    <li><span class="code">equalize</span>: <span class="code">TRUE</span> or <span class="code">FALSE</span>,
                        indicating whether or not to include this field to the Event Equalisation case.
                        The default value is FALSE (do not include)
                    </li>
                </ul>
                Model field contains:<br>
                two values - first and second model names.<br/>
                one value - when the stat values for one model are requested<br/>
                multiples pairs - for the aggregated scorecard<br/>
                Date range can be specified using 'fcst_valid_beg' or 'fcst_init_beg' and contains ine or more groups of
                two
                values - start and end dates. This example will create a scorecard for two periods (07-01 00:00 - 07-03
                12:00) and (07-05 00:00 - 07-10 12:00) <br/>
                <textarea rows="7" cols="100" style="border:none; font-size: 12px;">
        <field name="fcst_valid_beg">
          <val name="2011-07-01 00:00:00"/>
          <val name="2011-07-03 12:00:00"/>
          <val name="2011-07-05 00:00:00"/>
          <val name="2011-07-10 12:00:00"/>
        </field>
        </textarea>
                <br/>
                Init hour can contain one or more values. Date/time fields could have <span class="code">&lt;label&gt;</span> attribute. It wil bw used in the title instead on the list of dates.<br>
                For example, if 'fcst_valid_beg' example above had a label  <span class="code">&lt;field name="fcst_valid_beg" label="July 2011"&gt;</span>
                Then the title would be <br/>
                <div class="title1">METviewer Scorecard</div><div class="title2">for PR4RN_1405 and GFS2016</div><div class="title2">July 2011</div>
                <ul>
                    <li><span class="code">&lt;field</span> name="model"<span class="code">&gt;</span></li>
                    <ul>
                        <li><span class="code">&lt;val</span> name="<span class="loc">first_model</span>"
                            <span class="code">/&gt;</span></li>
                        <li><span class="code">&lt;val</span> name="<span class="loc">second_model</span>"
                            <span class="code">/&gt;</span></li>
                    </ul>
                    <li><span class="code">&lt;/field&gt;</span></li>
                    <li><span class="code">&lt;field</span>
                        name="fcst_valid_beg"<span class="code">&gt;</span></li>
                    <ul>
                        <li><span class="code">&lt;val</span> name="<span class="loc">start_date</span>"
                            <span class="code">/&gt;</span></li>
                        <li><span class="code">&lt;val</span> name=<span class="loc">"end_date</span>"
                            <span class="code">/&gt;</span></li>
                        optional second date range:
                        <li><span class="code">&lt;val</span> name="<span class="loc">start_date</span>"
                            <span class="code">/&gt;</span></li>
                        <li><span class="code">&lt;val</span> name=<span class="loc">"end_date</span>"
                            <span class="code">/&gt;</span></li>
                        or optional single date:
                        <li><span class="code">&lt;val</span> name="<span class="loc">date</span>"
                            <span class="code">/&gt;</span></li>

                    </ul>
                    <li><span class="code">&lt;/field&gt;</span></li>
                    <li><span class="code">&lt;field</span> name="init_hour"<span class="code">&gt;</span>
                    </li>
                    <ul>
                        <li><span class="code">&lt;val</span> name="<span class="loc">first_hour</span>"
                            <span class="code">/&gt;</span></li>
                    </ul>
                    <li><span class="code">&lt;/field&gt;</span></li>
                </ul>
            </li>
            <br/>

            <a name="rows"></a>
            <li><span class="code">&lt;rows&gt;</span>: describes an individual row in the scorecard
                table.<br/>
                Contains one or more nested <span class="code">&lt;field&gt;</span> elements
                <ul>
                    <li><span class="code">&lt;field&gt;</span>: describes an individual row or row group in
                        the scorecard table. <b>Attributes:</b>

                    </li>
                    <ul>
                        <ul style="list-style-image:url('att_bullet.gif')">
                            <li><span class="code">name</span>: database field name</li>
                        </ul>
                        <li><span class="code">&lt;val&gt;</span>: describes a value of the field. Field can
                            have more than one value element. <b>Attributes:</b>
                            <ul style="list-style-image:url('att_bullet.gif')">
                                <li><span class="code">name</span>: database field name</li>
                                <li><span class="code">label</span>: the label that shows up on the image</li>
                            </ul>
                        </li>
                        <li><span class="code">&lt;field&gt;</span>: next level database field name. Field can
                            have more than one field element
                        </li>
                    </ul>
                    <br/>
                </ul>
            </li>

            <a name="columns"></a>
            <li><span class="code">&lt;columns&gt;</span>: describes an individual column in the scorecard
                table.<br/>
                Contains one or more nested <span class="code">&lt;field&gt;</span> elements
                <ul>
                    <li><span class="code">&lt;field&gt;</span>: describes an individual column or column
                        group in the scorecard table. <b>Attributes:</b>
                        <ul>
                            <ul style="list-style-image:url('att_bullet.gif')">
                                <li><span class="code">name</span>: database field name</li>
                            </ul>
                            <li><span class="code">&lt;val&gt;</span>: describes a value of the field. Field can
                                have more than one value element. <b>Attributes:</b>
                                <ul style="list-style-image:url('att_bullet.gif')">
                                    <li><span class="code">name</span>: database field name</li>
                                    <li><span class="code">label</span>: the label that shows up on the image</li>
                                </ul>
                            </li>
                            <li><span class="code">&lt;field&gt;</span>: next level database field name. Field can
                                have more than one field element
                            </li>
                        </ul>
                        <br/>
                </ul>
            </li>
            <br/>
            <li><a name="agg_stat"></a>
                <span class="code">&lt;agg_stat&gt;</span>: <span class="code">TRUE</span>(bootstrapping) or
                <span class="code">FALSE</span>(student-T/normal),
                indicating whether or not to create a scorecard
                using aggregated statistics.<br/>The default value is TRUE (use bootstrapping )
            </li>
            <li><a name="boot_repl"></a>
                <span class="code">&lt;boot_repl&gt;</span>: number of bootstrapping replications, use 1 for
                no bootstrapping. Used when <span class="code">&lt;agg_stat&gt;</span>
                is TRUE
            </li>
            <li><a name="boot_random_seed"></a>
                <span class="code">&lt;boot_random_seed&gt;</span>: value of Random.seed for bootstrapping
                to get a reproducible random results.
                Used when <span class="code">&lt;agg_stat&gt;</span> is TRUE
            </li>

            <li><a name="plot_stat"></a>
                <span class="code">&lt;plot_stat&gt;</span>: name of the aggregation
                statistic.<span class="code">median</span> or <span class="code">mean</span>
                The default value is median.
            </li>

            <a name="tmpl"></a>
            <li><span class="code">&lt;tmpl&gt;</span>:</li>
            <ul>
                <li><span class="code">&lt;data_file&gt;</span>: name of the R data.frame data file, inside
                    of the <span class="code">&lt;r_work&gt;</span> folder
                    specified above
                </li>
                <li><span class="code">&lt;plot_file&gt;</span>: name of the output image file, inside of
                    the <span class="code">&lt;plots&gt;</span> folder specified
                    above
                </li>
                <li><span class="code">&lt;title&gt;</span>: title of the scorecard displayed on the image.
                    Default value:
                    <span style="font-weight: bold;">Verification Scorecard</span></li>
            </ul>

        </ul>
    </li>
</ul>
<br/>
<p>
    Here is a simple example. It produces <a class="loc" href="scorecard_sum_emc.html">this</a>
    scorecard :
    <br/><br/>
    <textarea rows="40" cols="100" style="border:none; font-size: 12px;">
  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
        <plot_spec>
    <connection>
      <host>db_host:3306</host>
      <database>database_name</database>
      <user>user_name</user>
      <password>password</password>
      <management_system>mariadb</management_system>
    </connection>

    <rscript>/usr/local/bin/Rscript</rscript>
    <folders>
      <r_tmpl>path_to_R_tmpl</r_tmpl>
      <r_work>path_to_R_work/</r_work>
      <plots>path_to_plots_folder</plots>
      <data>path_to_data_folder</data>
      <scripts>path_to_scripts_folder</scripts>
    </folders>
    <plot>
      <view_value>false</view_value>
      <view_symbol>true</view_symbol>
      <view_legend>false</view_legend>

      <stat_flag>EMC</stat_flag>
      <printSQL>FALSE</printSQL>

      <stat>DIFF_SIG</stat>
      <threshold_file>path_to_threshold_file</threshold_file>

      <left_column_names>
        <val></val>
        <val></val>
        <val></val>
      </left_column_names>

      <template>scorecard.R_tmpl</template>

      <plot_fix>
        <field name="model">
          <val name="PR4RN_1405"/>
          <val name="GFS2016"/>
        </field>
        <field name="fcst_valid_beg">
          <val name="2014-05-20 00:00:00"/>
          <val name="2014-07-30 00:00:00"/>
        </field>
        <field name="init_hour">
          <val name="00"/>
          <val name="12"/>
        </field>
      </plot_fix>

      <rows>
        <field name="stat">
          <val name="ANOM_CORR" label="Anom Corr"/>
          <field name="fcst_var">
            <val name="HGT" label="Heights"/>
            <field name="fcst_lev">
              <val name="P250"/>
              <val name="P500"/>
              <val name="P700"/>
              <val name="P1000"/>
            </field>
          </field>
        </field>

        <field name="stat">
          <val name="VL1L2_CORR" label="Anom Corr"/>
          <field name="fcst_var">
            <val name="WIND" label="Vector Wind"/>
            <field name="fcst_lev">
              <val name="P250"/>
              <val name="P500"/>
              <val name="P850"/>
            </field>
          </field>
        </field>


        <field name="stat">
          <val name="ANOM_CORR" label="Anom Corr"/>
          <field name="fcst_var">
            <val name="T" label="Temp"/>
            <field name="fcst_lev">
              <val name="P250"/>
              <val name="P500"/>
              <val name="P850"/>
            </field>
          </field>
        </field>

        <field name="stat">
          <val name="ANOM_CORR" label="Anom Corr"/>
          <field name="fcst_var">
            <val name="PMSL" label="MSLP"/>
            <field name="fcst_lev">
              <val name="MSL"/>
            </field>
          </field>
        </field>


        <field name="stat">
          <val name="RMSE"/>
          <field name="fcst_var">
            <val name="HGT" label="Heights"/>
            <field name="fcst_lev">
              <val name="P10"/>
              <val name="P20"/>
              <val name="P50"/>
              <val name="P100"/>
              <val name="P200"/>
              <val name="P500"/>
              <val name="P700"/>
              <val name="P850"/>
              <val name="P1000"/>
            </field>
          </field>
        </field>

        <field name="stat">
          <val name="VL1L2_RMSE" label="RMSE"/>
          <field name="fcst_var">
            <val name="WIND" label="Vector Wind"/>
            <field name="fcst_lev">
              <val name="P10"/>
              <val name="P20"/>
              <val name="P50"/>
              <val name="P100"/>
              <val name="P200"/>
              <val name="P500"/>
              <val name="P700"/>
              <val name="P850"/>
              <val name="P1000"/>
            </field>
          </field>
        </field>

        <field name="stat">
          <val name="RMSE"/>
          <field name="fcst_var">
            <val name="T" label="Temp"/>
            <field name="fcst_lev">
              <val name="P10"/>
              <val name="P20"/>
              <val name="P50"/>
              <val name="P100"/>
              <val name="P200"/>
              <val name="P500"/>
              <val name="P700"/>
              <val name="P850"/>
              <val name="P1000"/>
            </field>
          </field>
        </field>

        <field name="stat">
          <val name="ME" label="Bias"/>
          <field name="fcst_var">
            <val name="HGT" label="Heights"/>
            <field name="fcst_lev">
              <val name="P10"/>
              <val name="P20"/>
              <val name="P50"/>
              <val name="P100"/>
              <val name="P200"/>
              <val name="P500"/>
              <val name="P700"/>
              <val name="P850"/>
              <val name="P1000"/>
            </field>
          </field>
        </field>

        <field name="stat">
          <val name="VL1L2_BIAS" label="Bias"/>
          <field name="fcst_var">
            <val name="WIND" label="Wind Speed"/>
            <field name="fcst_lev">
              <val name="P10"/>
              <val name="P20"/>
              <val name="P50"/>
              <val name="P100"/>
              <val name="P200"/>
              <val name="P500"/>
              <val name="P700"/>
              <val name="P850"/>
              <val name="P1000"/>
            </field>
          </field>
        </field>

        <field name="stat">
          <val name="ME" label="Bias"/>
          <field name="fcst_var">
            <val name="T" label="Temp"/>
            <field name="fcst_lev">
              <val name="P10"/>
              <val name="P20"/>
              <val name="P50"/>
              <val name="P100"/>
              <val name="P200"/>
              <val name="P500"/>
              <val name="P700"/>
              <val name="P850"/>
              <val name="P1000"/>
            </field>
          </field>
        </field>

      </rows>

      <columns>
        <field name="vx_mask">
          <val name="G2/PNA" label="N.American"/>
          <field name="fcst_lead">
            <val name="24" label="Day 1"/>
            <val name="72" label="Day 3"/>
            <val name="120" label="Day 5"/>
            <val name="144" label="Day 6"/>
            <val name="192" label="Day 8"/>
            <val name="240" label="Day 10"/>
          </field>
        </field>

        <field name="vx_mask">
          <val name="G2/NHX" label="N.Hemisphere"/>
          <field name="fcst_lead">
            <val name="24" label="Day 1"/>
            <val name="72" label="Day 3"/>
            <val name="120" label="Day 5"/>
            <val name="144" label="Day 6"/>
            <val name="192" label="Day 8"/>
            <val name="240" label="Day 10"/>
          </field>
        </field>
        <field name="vx_mask">
          <val name="G2/SHX" label="S.Hemisphere"/>
          <field name="fcst_lead">
            <val name="24" label="Day 1"/>
            <val name="72" label="Day 3"/>
            <val name="120" label="Day 5"/>
            <val name="144" label="Day 6"/>
            <val name="192" label="Day 8"/>
            <val name="240" label="Day 10"/>
          </field>
        </field>
        <field name="vx_mask">
          <val name="G2/TRO" label="Tropics"/>
          <field name="fcst_lead">
            <val name="24" label="Day 1"/>
            <val name="72" label="Day 3"/>
            <val name="120" label="Day 5"/>
            <val name="144" label="Day 6"/>
            <val name="192" label="Day 8"/>
            <val name="240" label="Day 10"/>
          </field>
        </field>


      </columns>

      <agg_stat>false</agg_stat>
      <boot_repl>1000</boot_repl>
      <boot_random_seed>1</boot_random_seed>

      <tmpl>
        <data_file>scorecard.data</data_file>
        <plot_file>scorecard.png</plot_file>
        <title>METviewer Scorecard</title>
      </tmpl>

    </plot>
  </plot_spec>
  </textarea>
</p>

<br/>
<p>
    Here is an example of threshold configuration XML file that is used in <a
        class="loc" href="scorecard_sum_emc.html"> this </a>scorecard. Use Unicode characters as
    symbols (&amp;#9650; - &#9650;triangle) :
    <br/><br/>
    <textarea rows="40" cols="100" style="border:none; font-size: 12px;">
   <ranges>
     <legend_range>
       <symbol>&#9650;</symbol>
       <color>#009120</color>
       <background>#FFFFFF</background>
       <lower_limit>0.999</lower_limit>
       <upper_limit>1</upper_limit>
       <include_lower_limit>TRUE</include_lower_limit>
       <include_upper_limit>TRUE</include_upper_limit>
       <format_string>%s is better than %s at the 99.9%% significance level</format_string>
     </legend_range>

     <legend_range>
       <symbol>&#9652;</symbol>
       <color>#009120</color>
       <background>#FFFFFF</background>
       <lower_limit>0.99</lower_limit>
       <upper_limit>0.999</upper_limit>
       <include_lower_limit>TRUE</include_lower_limit>
       <include_upper_limit>FALSE</include_upper_limit>
       <format_string>%s is better than %s at the 99%% significance level</format_string>
     </legend_range>

     <legend_range>
       <color>#000000</color>
       <background>#A9F5A9</background>
       <lower_limit>0.95</lower_limit>
       <upper_limit>0.99</upper_limit>
       <include_lower_limit>TRUE</include_lower_limit>
       <include_upper_limit>FALSE</include_upper_limit>
       <format_string>%s is better than %s at the 95%% significance level</format_string>
     </legend_range>

     <legend_range>
       <color>#000000</color>
       <background>#BDBDBD</background>
       <lower_limit>-0.95</lower_limit>
       <upper_limit>0.95</upper_limit>
       <include_lower_limit>FALSE</include_lower_limit>
       <include_upper_limit>TRUE</include_upper_limit>
       <format_string>%s is worse than %s at the 95%% significance level</format_string>
     </legend_range>

     <legend_range>
       <color>#000000</color>
       <background>#F5A9BC</background>
       <lower_limit>-0.99</lower_limit>
       <upper_limit>-0.95</upper_limit>
       <include_lower_limit>FALSE</include_lower_limit>
       <include_upper_limit>TRUE</include_upper_limit>
       <format_string>%s is worse than %s at the 95%% significance level</format_string>
     </legend_range>

     <legend_range>
       <symbol>&#9662;</symbol>
       <color>#FF0000</color>
       <background>#FFFFFF</background>
       <lower_limit>-0.999</lower_limit>
       <upper_limit>-0.99</upper_limit>
       <include_lower_limit>FALSE</include_lower_limit>
       <include_upper_limit>TRUE</include_upper_limit>
       <format_string>%s is worse than %s at the 99%% significance level</format_string>
     </legend_range>

     <legend_range>
       <symbol>&#9660;</symbol>
       <color>#FF0000</color>
       <background>#FFFFFF</background>
       <lower_limit>-1</lower_limit>
       <upper_limit>-0.999</upper_limit>
       <include_lower_limit>TRUE</include_lower_limit>
       <include_upper_limit>TRUE</include_upper_limit>
       <format_string>%s is worse than %s at the 99.9%% significance level</format_string>
     </legend_range>

     <legend_range>
       <color>#000000</color>
       <background>#58ACFA</background>
       <include_lower_limit>FALSE</include_lower_limit>
       <include_upper_limit>FALSE</include_upper_limit>
       <format_string>Not statistically relevant</format_string>
     </legend_range>

   </ranges>
   </textarea>




<p>
    Here is an example of weights configuration XML  :
    <br/><br/>
    <textarea rows="40" cols="100" style="border:none; font-size: 12px;">
   <WeightRequirements name ="cbd_score">

<WeightToColor weight="6.4" color="#091534" />
<WeightToColor weight="3.2" color="#012998" />
<WeightToColor weight="2.4" color="#0145FE" />
<WeightToColor weight="2.0" color="#678FFE" />
<WeightToColor weight="1.2" color="#DBE5FF" />

	<Weight weight="6.4" >
	    <Criteria field="fcst_var" value="PRMSL" />
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>

	<Weight weight="6.4" >
		<Criteria field="fcst_var" value="PRMSL" />
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="6.4" >
	    <Criteria field="fcst_var" value="PRMSL" />
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>

	<Weight weight="6.4">
		 <Criteria field="fcst_var" value="PRMSL" />
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>
	<Weight weight="6.4">
		<Criteria field="fcst_var" value="PRMSL" />
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>

	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>
	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>

	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>

	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>


	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>
	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>
	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>
	<Weight weight="2.4" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="nhem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>

	<Weight weight="2.0" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P850" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>
	<Weight weight="2.0" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P850" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="2.0" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P850" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>
	<Weight weight="2.0" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P850" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>
	<Weight weight="2.0" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P850" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>

	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="trop" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>

	<Weight weight="3.2" >
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>
	<Weight weight="3.2" >
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="3.2" >
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>
	<Weight weight="3.2" >
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>
	<Weight weight="3.2" >
		<Criteria field="fcst_lev" value="L0" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>

	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="HGT" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P500" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>

	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="240000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="480000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="720000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="960000" />
	</Weight>
	<Weight weight="1.2" >
		<Criteria field="fcst_var" value="WIND" />
		<Criteria field="vx_mask" value="shem" />
		<Criteria field="fcst_lev" value="P250" />
		<Criteria field="fcst_lead" value="1200000" />
	</Weight>


</WeightRequirements>
   </textarea>
</body>
</html>
